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COPYRIGHT NOTICE 
[0001] A portion of the disclosure of this patent document contains material 

which is subject to copyright protection. The copyright owner has no objection to the 
facsimile reproduction by anyone of the patent document or the patent disclosure, as it 
appears in the Patent and Trademark Office patent file or records, but otherwise reserves 
all copyright rights whatsoever. 

CROSS-REFERENCE TO RELATED APPLICATIONS 
[0002] This application is related to the following co-pending applications which 

are each hereby incorporated by reference in their entirety: 

[0003] SYSTEMS FOR CONTROLLING VIDEO OR MOTION CAMERAS, 

U.S. Application No.09/391,141, Inventors: Jon Foote, et al., filed on September 7, 1999. 

(Attorney's Docket No. FXPAL-IP-00-008) 

[0004] COMPUTER-ASSISTED MEETING CAPTURING SYSTEM, U.S. 

Application No. 09/981,735, Inventors: Shingo Uchihashi, et al., filed on February 16, 
2001. (Attorney's Docket No. FXPAL-IP-00-013) 

[0005] VIDEO PRODUCTION THROUGH COLLABORATIVE CONTROL 

OF FLYSPECS, U.S. Application No. 10/205,739, hiventors: Qiong Liu, et al., filed on 
July 26, 2002. (Attorney's Docket No. FXPAL-IP-00-016) 
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FIELD OF THE DISCLOSURE 



[0005] 



The present invention disclosure relates to a graphical computer interface 



which allows one or more users to interact with one or more real-world devices via by 
interacting with video images of the devices and their environment. 



one extreme, a person can interact with it directly (e.g., push a door to open it). At 
another extreme, a person can interact with the real world through various levels of 
indirection such as with a control panel or a software program. Although direction 
interaction with the real world is probably the most intuitive way for a person to affect 
changes, sometimes such interaction is not possible due to dangerous conditions (e.g., 
manipulation of nuclear fuel), problems of scale (e.g., manipulation of the interior of a 
single cell organism), and problems of distance (e.g., teleconferencing). 
[0007] Teleconferencing over the Intemet is increasingly popular for many 

applications, such as distance leaming. Through remote participation, students can attend 
classes from their dorm rooms, scientists can participate in seminars held in other 
coimtries, and executives can discuss critical issues without leaving their offices. There 
are many ways to support such activities: one-way audio broadcast, two-way audio 
communication, and two-way video communication. However, each is limited in the 
amount of intuitive interaction afforded to the participants. 



BACKGROUND 



[0006] 



Interacting with the real world can be accomplished in various ways. At 



BRIEF DESCRIPTION OF THE DRAWINGS 



[0008] 



Figure 1 is an illustration of a graphical user interface (GUI) in one 



embodiment. 



[0009] 



Figure 2 is an illustration of selection of a zoom region in one 



embodiment. 



[0010] 
[0011] 
[0012] 



Figure 3 is an illustration of device "hot spots" in an embodiment 
Figure 4 is an illustration of a pop-up control panel in one embodiment. 
Figure 5 is an illustration of drag and drop possibiUties in one 



embodiment. 
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[0013] Figure 6 is an illustration of annotations in one embodiment. 

[0014] Figure 7 is an illustration of a system for a shared interactive environment 

in one embodiment. 

DETAILED DESCRIPTION 
[0015] The invention is illustrated by way of example and not by way of 

limitation in the figures of the accompanying drawings in which like references indicate 
similar elements. It should be noted that references to "an" or "one" embodiment in this 
disclosure are not necessarily to the same embodiment, and such references mean at least 
one. 

[0016] In the following description, various aspects of the present invention will 

be described. However, it will be apparent to those skilled in the art that the present 
invention may be practiced with only some or all aspects of the present invention. For 
purposes of explanation, specific numbers, materials and configurations are set forth in 
order to provide a thorough understanding of the present invention. However, it will be 
apparent to one skilled in the art that the present invention may be practiced without the 
specific details. In other instances, well-known features are omitted or simplified in 
order not to obscure the present invention. 

[0017] Parts of the description will be presented in data processing terms, such as 

data, selection, retrieval, generation, and so forth, consistent with the manner commonly 
employed by those skilled in the art to convey the substance of their work to others 
skilled in the art. As well understood by those skilled in the art, these quantities take the 
form of electrical, magnetic, or optical signals capable of being stored, transferred, 
combined, and otherwise manipulated through electrical, optical, and/or biological 
components of a processor and its subsystems. 

[0018] Various operations will be described as multiple discrete steps in turn, in a 

manner that is most helpful in understanding the present invention, however, the order of 
description should not be construed as to imply that these operations are necessarily order 
dependent. 

[0019] Various embodiments will be illustrated in terms of exemplary classes 

and/or objects in an object-oriented programming paradigm. It will be apparent to one 
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skilled in the art that the present invention can be practiced using any number of different 
classes/objects, not merely those included here for illustrative purposes. Furthermore, it 
will also be apparent that the present invention is not limited to any particular software 
programming language or programming paradigm. 

[0020] Figure 1 is an illustration of a graphical user interface (GUI) in one 

embodiment. Window 100 displays two live video images of the same physical location 
(e.g., a conference room). Li one embodiment, a live video image may be updated at a 
low frame rate due to network bandwidth limitations, for example. In another 
embodiment, a live video image may only be updated as necessary to reflect any changes 
in the image. In yet another embodiment, a live video image consists of a stream of delta 
images that represents changes to an original image. 

[0021] For purposes of clarity, video images are depicted in this illustration with 

simple drawings rather than with bitmaps of actual video signals. The drawings 
emphasize the physical devices that would be visible from the video images. The upper 
window 102 displays a panoramic video view of the location whereas the lower window 
104 presents a close-up (or zoom) video view of a region 106 in the panoramic view. 
The zoom region can be any region in the panoramic view. The smaller the zoom region, 
the greater the magnification will be in the zoom view. In another embodiment, the zoom 
region can also be in the zoom view. In this case, the zoom view can be fiuther 
magnified to show only the zoom region. In the location shown by the views, a number 
of devices are visible. A device can be controlled and/or manipulated over a 
communication channel (e.g., IEEE 802.11, Internet, Bluetooth®, Infrared, etc.) By way 
of a non-limiting illustration, panoramic view 102 contains two displays (108 and 110), a 
portable computer 112 and a printer 114. There may also be other devices, such as 
speakers, microphones, lights, etc. Users (or participants) can interact with these physical 
devices through use of the GUI, as will be illustrated. 

[0022] By way of a non-limiting example, the GUI can run as an application in a 

web browser, a stand-alone application or as a combination of the two. In one 
embodiment, the GUI can be implemented as a Java^^ applet. The Java programming 
language and run-time environment are available from Sun Microsystems, Inc. of San 
Jose, California. The GUI can run on any computing device that can connect to a 
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network such as, and by way of a non-limiting-example, the Internet. By way of a further 
non-limiting example, the device could be a personal computer, a laptop or tablet 
computer, a personal data assistant (PDA), a mobile phone, a personal communication 
device, a wearable computing device, etc. Display devices can include plasma poster 
boards, projectors, video monitors, televisions, image projection systems, electronic 
whiteboards, etc. 

[0023] The video streams are provided to the views (102, 104) by a special 

camera system (hereinafter referred to as "FlySPEC") that includes a panoramic camera 
and a pan-tilt-zoom (PTZ) camera. The panoramic camera is fixed and provides a 
panoramic view of a location whereas the PTZ camera is able to pan, tilt and zoom which 
enables it to magnify portions of the location beyond the resolution of the panoramic 
camera. In one embodiment, the panoramic camera is constructed with two normal fixed 
cameras, and the PTZ camera is mounted on top of the panoramic camera. By locating 
the PTZ camera close to the panoramic camera, the coordinate system used by the PTZ 
camera can be approximated with the coordinate system used by the panoramic camera 
when a device of interest is relatively far away from the cameras. With such coordinate 
approximation, there is no need for a complicated coordinate transformation from the 
panoramic camera to the PTZ camera for purposes of moving the PTZ camera. In 
another embodiment, both cameras can have independent coordinates systems and a 
function can be provided to map from one system to the other. 

[0024] In one embodiment, the FlySPEC panoramic camera can cover about 100° 

field-of-view with two fixed cameras. Each fixed camera can have a 58° field of view 
and a Va' 512 by 492 color CCD sensor. The PTZ camera can have a 48.8° filed-of-view, 
a 1/3" 768 by 492 color CCD sensor, and 12X optical zoom. Since the PTZ camera can 
pan from -100° to +100°, and tilt from -25° to +25°, it can move to all positions covered 
by the panoramic system. Since the smallest filed of view for the PTZ camera is about 
4.4°, the maximum magnification that the PTZ camera can achieve in one embodiment is 
approximately 

ji^ 58x768 

Magnification^ = = 19.77 

4.4x512 

This means that a 39 x 25 image captured by the panoramic camera can be enlarged to a 
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768 X 492 image by the PTZ cam. In another embodiment, a higher magnification PTZ 
camera can be employed. 

[0025] The present disclosure is not limited to having a single FlySPEC in a 

single location. Furthermore, the present disclosure contemplates multiple users 
accessing a given FlySPEC from different GUI's on different devices. When more than 
one user is utilizing a FlySPEC, conflicts may arise regarding use of the PTZ camera 
since different users may want to focus on different zoom regions. In one embodiment, a 
management algorithm coordinates conflicting camera commands. Commands from 
users are separated into groups of similar view requests. The PTZ camera can zoom in 
and allow users to see small devices clearly. On the other hand, the panoramic camera 
can provide different video streams to multiple users simultaneously, and can provide 
reasonable video for a wide field of view. A coordination algorithm uses the PTZ camera 
to serve the group with the most demanding (i.e., requesting highest zoom-in) image 
requests, and scales the panoramic camera image stream to serve general uses who do not 
need as much resolution. 

[0026] Referring again to Fig. 1, there are user-selectable controls available for 

allowing a user to cycle through FlySPEC cameras within a given location, and to change 
locations. In one embodiment, the physical location of the FlySPEC in use by the GUI is 
evident from the window 100 title bar ( "Conference Room B*') as is the FlySPEC within 
the location ("View 1"). In another embodiment, a FlySPEC camera can be mounted on 
a mobile platform, such as a robot or other device, so that it can be repositioned within a 
given location, or change locations entirely. User-selectable control 116 will allow the 
user to change the physical location. By way of a non-limiting illustration, it may do so 
by providing the user with an interactive map of all potential locations. It may also allow 
the user to choose a FlySPEC within the new location as part of the location selection 
process. Alternatively, the GUI can connect to a FlySPEC in the new location that has 
been designated as the default or initial view. User-selectable control 118 allows the user 
to change FlySPEC cameras within a given location (if there is more than one) thus 
allowing the user to cycle through each FlySPEC until a desired view is achieved. 
[0027] A user interacting with the GUI can choose the zoom region using any of a 

number of methods. Figure 2 is an illustration of selection of a zoom region in one 
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embodiment. The zoom region can be determined by using the end points of a diagonal 
line 202 drawn by the user in either view to serve as opposite comers of a bounding 
rectangle 204. The bounding rectangle 204 becomes the zoom region. The user can 
draw the diagonal line (which does not have to be straight) using any nxmiber of input 
devices, including but not limited to a mouse, a pen, a finger, voice commands, hand 
gesture, etc. If line drawn in the panoramic view 102, then the resulting zoom region is 
presented in zoom view 104. If a line is drawn in the zoom view, then that zoom view is 
further magnified to present just the region the user selected. In another embodiment, the 
user can select icon 200 which will set the zoom region to be centered around a mouse 
cursor (and having a default magnification) if the user clicks a mouse button while the 
cursor is one of the views. For example, if the user has selected icon 200 and then clicks 
a mouse button while the mouse cursor is in the panoramic view, the zoom region will 
occupy a region (of default size) centered around the point at which the mouse cursor was 
at the time of the click. It is contemplated that any input device (including those not yet 
conceived) used to select and manipulate devices in the video stream(s) is within the 
scope and spirit of this disclosure. 

[0028] In addition to merely viewing devices in the panoramic view 102 and the 

zoom view 104, a user can interact with devices if the devices have "hot spots" defined in 
one or more views. Figure 3 is an illustration of device "hot spots" in an embodiment. 
A hot spot is a region of a view that intersects a device in the video image stream (e.g., 
display, printer, scanner, speaker, computer, light control, etc.). Devices with hot spots 
can be manipulated through GUI 100. Hot spots in Fig. 3 are depicted for illustrative 
purposes by dashed rectangles. By way of a non-limiting illustration, when a user 
positions a mouse cursor (or some other input device) over a hot spot, an embodiment 
makes the user aware of the hot spot by automatically changing the default cursor to an 
intuitive icon (e.g., a printer icon to indicate a printer device). Instead of or in addition to 
changing the mouse cursor to an icon, call out boxes can be displayed indicating the 
name, type of device, and/or device status. For example, there are two plasma display 
boards in Fig. 3 having call outs "Plasma Display 1" and "Plasma Display 2", a fax 
machine ("Joe's Fax Machine"), a laptop computer ("Sue's Laptop") and a printer 
("Printer 1 (off line)"). In one embodiment, a user can force all hot spots and/or call outs 
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to appear by selecting icon 300. 

[0029] Device hot spots can be defined statically or dynamically. In one 

embodiment, static hot spots can be defined with a tool that allows a user to specify a 
bounding region in a view (102 or 104) and associate device attributes with that region. 
By way of a non-limiting example, in Fig. 1 a user could specify a hot spot for printer 
device 114 by drawing a rectangle around it in view 102. After doing so, an input 
window could appear to allow the user to specify the printer's attributes. In this example, 
such information could include device type (printer), make, model, network address (e.g., 
uniform resource locator, Internet Protocol address, etc.), and GUI behaviors. GUI 
behaviors can include the following: 

■ whether the mouse cursor (or other input device cursor) will change when it is 
placed over device's hot spot and if so, what it should change to; 

■ whether there is a call out box associated with the device and if so, what 
information it will contain; 

■ whether the device can accept a "drag" operation; 

■ whether the device can accept a "drop" operation; and 

■ whether the device has a pop-up control panel and if so, the contents of the 
control panel. 

[0030] A device's GUI behaviors as well as its attributes can be naturally 

represented as a class in an objected-oriented software paradigm. In one embodiment, 
devices can be defined in a class hierarchy wherein more specific device t>pes can inherit 
attributes and behaviors from less specific ones. By way of a non-limiting example, a 
"Lexmark E323" printer class could inherit from a general "LaserPrinter" class. Besides 
the aforementioned GUI behaviors, a device class's methods could also include handlers 
for GUI events such as when a mouse cursor enters or exits the device's hot spot, mouse 
drag, mouse drop, mouse button activity, etc. Device definitions can be stored in a 
device database. 

[0031] In one embodiment, dynamically defined hot spots can arise when a 

device with a radio transmitter attached to it emits a signal that is used to locate the hot 
spot for the device and also identify the device. By way of a non-limiting example, such 
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a signal could conform to IEEE 802.11, Bluetooth®, or other wireless standards. If a 
given location has a radio direction finder (RDF) or similar device, devices with 
transmitters can have hot spots assigned to them dynamically by centering a hot spot 
region around the source of the transmission. Alternatively, the transmission itself could 
carry information pertaining to the location of the device. In one embodiment, the RDF 
can be located on a FlySPEC. The transmission can also identify the device either as a 
predefined device that exists in the device database or as a generic device. If it is a 
generic device, a device class that closely models its attributes and behaviors can be 
created for it dynamically. In another embodiment, the location of a device can be 
determined based on acoustic signal. A commercially available acoustic location finder 
is the IS-600 Mark 2 Precision Motion Tracker available from InterSense (Burlington, 
Massachusetts). In either case, the device class can have information regarding the 
physical dimensions of the device and this can be used together with the location 
information to define the bounding region of the device's hot spot. If the transmitted 
signal does not provide the device^s location, the device's distance from the RFD can be 
estimated based on the strength of the transmission signal or, in another embodiment, by 
using radar. 

[0032] In one embodiment, dynamically defined hot spots can arise through 

image pattern recognition of devices within the view of a FlySPEC. If the video streams 
coming out of the FlySPEC are analyzed for shapes that designate devices (e.g., printers, 
scanners, video monitors, etc.), these devices can be automatically added to the device 
database by creating device classes on the fly to model the attributes and behaviors of the 
devices so recognized. In another embodiment, if a dynamically recognized device has 
an IR (infrared) interface (as many audio/visual devices do), its device class can include a 
behavior to render a control panel on the GUI. The control panel can allow GUI users to 
configure/control the device by sending it IR commands. Such an embodiment 
contemplates including an IR transmitter in same location as the device. Optionally, the 
IR transmitter can be part of a FlySPEC. In another embodiment, a FlySPEC can be 
mounted on a mobile platform, such as a robot, so that it can be repositioned within a 
given location, or change locations entirely. Such a configuration would allow the 
FlySPEC to roam and control devices as it encountered them. 
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[0033] One way for a user to interact with a device in the GUI is through a 

device-specific control panel. Figure 4 is an illustration of a pop-up control panel in one 
embodiment. As part of a device's behavior methods, a device can have a method that 
renders a control panel in response to a request by the GUI. By way of a non-limiting 
example, this can be trigged by the user selecting a mouse button while the mouse cursor 
is in a device's hot spot, through a voice command, through a mouse or hand gesture, etc. 
By way of a non-limiting illustration, plasma display board 400 has a control panel 402 
that allows a user to control the display be interacting with the video image stream in 
view 102. The control panel includes an on/off button 404 to turn the display on or off, a 
contrast button 406 to adjust the display's contrast, and a menu button 408 to access other 
display functions. All devices can have such pop-up control panels, which may 
themselves include many nested levels of GUI. Since multiple users may utilize the GUI 
on multiple computing devices, changes made to devices (e.g., via a control panel) by one 
user will be reflected in the views of all other users. 

[0034] In another embodiment, users can interact with real world devices 

represented in video image streams by manipulating device information through a drag- 
and-drop metaphor. Drag and drop (D&D) works by a user selecting a source device in a 
GUI (e.g., by holding a mouse button down when the mouse cursor is over the device), 
"dragging" a representation (e.g., icon, image, etc.) of the device to another location in 
the GUI (e.g., by keeping the mouse button pressed down while moving the mouse cursor 
to the new location), and then "dropping" the representation (usually by releasing the 
mouse button) on a destination device. D&D is not Umited to being accomplished by 
solely by mouse gestures. It could also be accomplished through voice commands, hand 
gestures, other mouse gestures, human eye tracking, etc. It will be appreciated that the 
present disclosure is not limited to a single user input device or method of interacting 
with a GUI. 

[0035] Figure 5 is an illustration of drag and drop possibilities in one 

embodiment. The nonlinear lines in Fig. 5 illustrate possible drag and drop sources and 
destinations by connecting them to each other, but do not indicate or mandate any 
particular order of operations or any particular technique for manipulating an input device 
such as a mouse. Local icon 500 represents a file and/or information that exists on a 
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user's computing device, and is visible to the user along with GUI 100. In one 
embodiment, the icon is visible on the computing device's desktop metaphor or other 
GUI. A user can drag the source file/information 500 and drop it onto a device in one of 
the video views 102 or 104 (e.g., destination displays 108 or 110, printer 114, and/or 
computing device 112). When a user attempts to drop the file/information onto the 
aforementioned devices, the GUI can query the device class to see if it has a behavior 
method that can accept the "drop" request. If so, the drop operation is allowed. If not, 
the mouse cursor could change to reflect the fact that device is not a drop target (e.g., the 
mouse cursor could change to an "X"). If the device can accept the drop request, the 
file/information 500 is actually transferred (i.e., copied or moved) to the device. By way 
of a non-limiting example, if the destination device is a display (502, 506, 508), the 
file/information can be displayed on the device. If the destination device is a printer 
(510), the file/information can be printed. If the destination device is computing device 
(504), the file/information can be transferred to the computing device. If the destination 
device is a mobile phone (not shown), the file/information is transferred to the phone. It 
will be appreciated that any device capable of commimicating over a communication 
channel - not those listed here merely for illustrative purposes - can potentially be part of 
such a D&D scheme. It will further be appreciated that a file/information can contain 
any type of digital information, including but not limited to images, sounds, documents, 
slide presentations, audio/visual presentations, and streaming media. Of course, when 
one user performs a D&D operation that causes a change in a view, that change will be 
evident to all other users who happen to be sharing that view (through other instances of 
the GUI). 

[0036] In addition to dragging files/information from the user's local computing 

device to devices that appear in the live video views, the opposite also holds true. A user 
can drag information from a device in a video view (102 or 104) to the user's local 
device's desktop metaphor or file system. During the operation, the mouse cursor can 
change to represent the type of file/information that is being transferred. By way of a 
non-limiting example, if the source of the D&D operation is a display device, the mouse 
cursor can be changed to a thumb-nail sketch of the current display on the display device. 
A file/information dragged from a display device and dropped on the user's computing 
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device (502, 506, 508) can contain a representation of the graphical information that was 
rendered on the display device at the time the D&D operation took place. In one 
embodiment, display devices (or processes that manage them) can implement image 
stacks. An image stack acts like a stack data structure (i.e., first-in, first-out) where the 
image on the top of the stack is the image that is actually displayed. When an file is 
"dropped" on a display, it is added to the top of the stack. When an image is "dragged" 
from a display, it is removed from the top of the stack and the new top of the stack is then 
displayed in its place. Again, when one user performs a D&D operation that causes a 
change in a view, that change will be evident to all other users who happen to be sharing 
that view (through other instances of the GUI). 

[0037] In addition to dragging files/information to and from a user's computing 

device, files or information can be dragged to/from a devices that appear in views. By 
way of a non-limiting example, files/information can be dragged between displays (512, 
518). In one embodiment, when a file/information is dragged from one display to 
another, the top of the source display device's image stack is removed and placed on the 
top of the destination display device's image stack. This has the effect of dragging an 
image from one display to another. Information/files can also be dragged between 
displays and computing devices (514, 516), between displays and printers (520, 524, 
522), and from computing devices to printers (530). Again, when one user performs a 
D&D operation that causes a change in a view, that change will be evident to all other 
users who happen to be sharing that view (through other instances of the GUI). 
[0038] In another embodiment, users can interact with devices and views by 

annotating them. Figure 6 is an illustration of annotations in one embodiment. In one 
embodiment, an annotation tool 604 can be provided to allow users to make annotations 
(e.g. notes, drawings, etc.) on a view (102, 104) and/or on a device as it appears in a 
view. When one user creates/changes/deletes an annotation, the result is visible to all 
other users who happen to be sharing that view (through other instances of the GUI). 
Annotation 600 is a hand-written name of a meeting participant ("John") that in one 
embodiment can be drawn with a mouse after selecting annotation tool 604 in GUI 100. 
Annotation 602 is an example of an annotation drawn on display device 108 as it appears 
in a view. In this case, a user has circled a portion of the graph being displayed in one of 
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the views (102, 104). If the display device (or its display management process) has 
implemented an image stack, the annotation can be applied to the image on the top of the 
stack (i.e., the currently displayed image). In one embodiment, if the aimotation even 
partially overlaps the display device, it will be applied in its entirety to the top of the 
device's image stack. In another embodiment, only the portion of the annotation that 
overlaps with the display device will be applied to the device's image stack. All other 
users viewing that device will see the annotation (including viewers who are actually 
present where the device is located). In addition, the annotation can also be visible in the 
zoom view 104 if it is encompassed by the zoom region. 

[0039] Figure 7 is an illustration of a system for a shared interactive 

environment in one embodiment. Although this diagram depicts processes as 
functionally separate, such depiction is merely for illustrative purposes. It will be 
apparent to those skilled in the art that the processes portrayed in Figure 7 can be 
arbitrarily combined or divided into separate software, firmware or hardware 
components. Furthermore, it will also be apparent to those skilled in the art that such 
devices, irregardless of how they are combined or divided, can execute on the same 
computing device or can be arbitrarily distributed among different computing devices 
connected by a network. 

[0040] Physical devices and the processes that enable manipulating them in a 

shared interactive environment can all be connected by one or more networks 700 (e.g., 
the Intemet). Devices (702-718) may be distributed in one or more separate physical 
locations. Each location that is part of the shared interactive environment that has one or 
more FlySPEC devices. FlySPEC devices may have attached video server hardware 
(702, 704) to improve the performance of delivering image streams to multiple users. In 
one embodiment, one or more FlySPEC Manager processes 720 configure and control the 
FlySPEC devices. In one embodiment, the FlySPEC manager process can capture video 
streams from fixed cameras (on FlySPEC device(s)) and stitch them into a panoramic 
video stream in real-time. Video streams from the FlySPEC devices and/or the FlySPEC 
manager(s) are provided to one or more end user computing devices (726, 734), 
depending on which locations and views are selected for viewing on each. Each end user 
computing device runs an instance of the GUI 100. One or more device controllers 722 
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are responsible for controlling the devices (702-718) based on ender user interaction with 
the GUI (e.g., through control panels, D&D, etc.). One or more device databases 724 can 
contain information (e.g., device classes) for each device (702-718), including devices 
that are dynamically added via image pattern recognition or RFD. In another 
embodiment, end user computing devices (726, 724) running instances of GUI 100 can 
be situated in a location and manipulated with other instances of GUI 100 executing on 
other devices, 

[0041] Device tracker 728 is responsible for keeping track of newly discovered 

devices (e.g., devices that are discovered through radio frequency transmission or image 
pattern recognition). The device tracker creates new classes for these devices, defines 
their hot spots, updates the device database(s), and informs any affected GUI's of the new 
hot spots so that the end users may then interact with the new devices. Display manager 
730 is responsible for managing image stacks for all display devices (708, 718) that 
utilize them. Device controller 722 is responsible for sending commands to devices (702- 
718) to carryout end user interactions, such as through popup control panels or D&D. 
[0042] One embodiment may be implemented using a conventional general 

purpose or a specialized digital computer or microprocessor(s) programmed according to 
the teachings of the present disclosure, as will be apparent to those skilled in the 
computer art. Appropriate software coding can readily be prepared by skilled 
programmers based on the teachings of the present disclosure, as will be apparent to 
those skilled in the software art. The invention may also be implemented by the 
preparation of integrated circuits or by interconnecting an appropriate network of 
conventional component circuits, as will be readily apparent to those skilled in the art. 
[0043] One embodiment includes a computer program product which is a storage 

medium (media) having instructions stored thereon/in which can be used to program a 
computer to perform any of the features presented herein. The storage medium can 
include, but is not limited to, any type of disk including floppy disks, optical discs, DVD, 
CD-ROMs, microdrive, and magneto-optical disks, ROMs, RAMs, EPROMs, 
EEPROMs, DRAMs, VRAMs, flash memory devices, magnetic or optical cards, 
nanosystems (including molecular memory ICs), or any type of media or device suitable 
for storing instructions and/or data. 
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[0044] Stored on any one of the computer readable medium (media), the present 

invention includes software for controlling both the hardware of the general 
purpose/specialized computer or microprocessor, and for enabling the computer or 
microprocessor to interact with a human user or other mechanism utilizing the results of 
the present invention. Such software may include, but is not limited to, device drivers, 
operating systems, execution environments/containers, and user applications. 
[0045] The foregoing description of the preferred embodiments of the present 

invention has been provided for the purposes of illustration and description. It is not 
intended to be exhaustive or to limit the invention to the precise forms disclosed. Many 
modifications and variations will be apparent to the practitioner skilled in the art. 
Embodiments were chosen and described in order to best describe the principles of the 
invention and its practical application, thereby enabling others skilled in the art to 
understand the invention, the various embodiments and with various modifications that 
are suited to the particular use contemplated. It is intended that the scope of the invention 
be defined by the following claims and their equivalents. 
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